Add plain-rag dense chunk RAG reference row to memory-bench (#21) - #91

What & Why

Contributes the plain-rag reference row requested in #21 ("Submit your system's result to memory-bench (mem0, Letta, Khoj, plain RAG, yours)").

plain-rag (bench/adapters/plain_rag.py, version: "reference", url: "https://huggingface.co/intfloat/multilingual-e5-base") runs the exact same markdown chunker (sqlite_graph_memory.index_notes, MAX_CHARS=1200, OVERLAP=200), intfloat/multilingual-e5-base bi-encoder (brain_ask.load_encoder('cpu'), brain_ask.query_sims, brain_ask.retrieve_candidates with topk_retrieve=60), and shared _extract.best_sentence answer floor as sqlite-graph-memory, without the 1-hop [[wikilink]] graph expansion (expand_1hop) and without the mmarco-mMiniLMv2 cross-encoder reranker (rerank_candidates).

Because .github/workflows/bench.yml already caches intfloat/multilingual-e5-base (hf-e5base-mmarco-v1) and installs torch==2.4.1 + sentence-transformers==3.1.1 on CPU, plain-rag requires zero additional dependencies or downloads in CI and is added to bench/ci_reproduced.txt (CI-reproduced: yes).

What This Row Isolates on the Leaderboard

Putting plain-rag right next to sqlite-graph-memory-vector (dense + cross-encoder reranker) and sqlite-graph-memory (dense + 1-hop graph + cross-encoder reranker) on the identical topk_retrieve=60 candidate pool isolates the exact contribution and cost of each stage on the 202-note vault:

1. Bi-encoder vs. cross-encoder on zero-overlap paraphrase: Pure e5-base cosine (plain-rag) retrieves 10/10 (1.00) paraphrase notes and 50% (0.50) of paraphrase_bridge notes in the top 5; the mmarco-mMiniLMv2 cross-encoder reranker demotes zero-shared-word paraphrases to 0.80 (vector) / 0.50 (graph) on paraphrase and 0.08 on paraphrase_bridge.
2. Where the cross-encoder and 1-hop graph win: Adding the cross-encoder lifts bridge R@5 from 0.87 (plain-rag) to 0.97 (sqlite-graph-memory-vector), and adding the 1-hop [[wikilink]] graph lifts bridge R@5 to 1.00 (sqlite-graph-memory), while the cross-encoder's < 0.0 logit boundary more than doubles abstention on unanswerable questions (0.18 $\to$ 0.45).

Verification

$ python bench/run.py --system plain-rag
class                  n    R@5   R@10     ans  cite_vb faithful  abstain  f_abst
absent_attribute      12    -      -       -        -      1.000    0.000     -  
absent_entity         10    -      -       -        -      1.000    0.400     -  
bridge                15  0.867  0.933   0.000    0.000    1.000      -     0.000
fact                  28  1.000  1.000   0.679    0.679    1.000      -     0.000
paraphrase            10  1.000  1.000   0.000    0.000    1.000      -     0.900
paraphrase_bridge      6  0.500  0.500   0.000    0.000    1.000      -     0.833
temporal              10  1.000  1.000   0.400    0.400    1.000      -     0.000
OVERALL               91  0.927  0.942   0.333    0.333    1.000    0.182   0.203

$ python bench/validate.py
ok   vault and questions match the generator
ok   bm25.json  (R@5 0.768, abstain 0.364, CI-reproduced: True)
ok   fidelis.json  (R@5 0.855, abstain 0.182, CI-reproduced: True)
ok   mem0-local.json  (R@5 0.870, abstain 0.136, CI-reproduced: True)
ok   plain-rag.json  (R@5 0.927, abstain 0.182, CI-reproduced: True)
ok   second-brain.json  (R@5 0.761, abstain 0.000, CI-reproduced: True)
ok   sqlite-fts5.json  (R@5 0.761, abstain 0.000, CI-reproduced: True)
ok   sqlite-graph-memory-vector.json  (R@5 0.884, abstain 0.455, CI-reproduced: True)
ok   sqlite-graph-memory.json  (R@5 0.848, abstain 0.455, CI-reproduced: True)
ok   tangleclaw-style.json  (R@5 0.768, abstain 0.455, CI-reproduced: True)
ok   LEADERBOARD.md matches the result files
VALID

$ python bench/validate.py --reproduce plain-rag --tolerance 0
reproduce plain-rag: worst metric drift 0.0000 (tolerance 0.0000); 0/91 questions changed (max 0)
ok   plain-rag reproduces

$ pytest -q
267 passed, 2 skipped in 3.41s

AI Disclosure: Built and verified by an AI coding agent fleet under human direction (testament receipt b60ab4bc47f8439d82ae33bffb397b12).

— TangleClaw'd by Jason Vaughan
Human-directed. Agent-executed. Evidence-backed.
TangleClaw.com